gh-158585: Optimize bytes object creation by vstinner · Pull Request #158664 · python/cpython

vstinner · 2026-10-03T16:46:01Z

Add bytes_alloc() helper function. bytes_alloc() and _PyBytes_FromStringAndSize() have less conditional branches than previous code, and so should be a little bit faster.

Rename existing bytes_alloc() to bytes_type_alloc().
Replace _PyBytes_FromSize(calloc=1) with _PyBytes_FromSizeZero().
Add set_ob_shash_unsafe(): similar to set_ob_shash() but don't use an atomic operation on Free Threading. Use this new function on newly allocated bytes objects and in bytes_resize_inplace().
Replace PyBytes_FromStringAndSize(NULL, size) with bytes_alloc(size).
Use bytes_alloc() in PyBytes_FromString() and _PyBytes_Repeat().
Add _PyBytes_FromStringAndSize(): similar to PyBytes_FromStringAndSize(), but str must not be NULL.
Replace PyBytes_FromStringAndSize() with _PyBytes_FromStringAndSize().
Test that PyBytes_FromString() and PyBytes_FromStringAndSize() return singletons for 0 or 1 bytes.

Issue: C API: Rework and enhance the PyBytesWriter implementation #158585

vstinner · 2026-10-03T17:02:12Z

Results of PR gh-158665 benchmark: Mean +- std dev: [ref] 34.8 ns +- 0.4 ns -> [change] 32.2 ns +- 0.5 ns: 1.08x faster.

The change makes PyBytesWriter_Finish() 1.08x faster, it saves 2.6 nanoseconds.

vstinner · 2026-10-03T17:22:30Z

Oh. A side effect of this change is that b'abc' * 0 now returns the empty string singleton. I added a test for that.

vstinner · 2026-10-03T17:39:07Z

I also ran a benchmark on PyBytes_FromStringAndSize("abc", 3) and PyBytes_FromString("abc"): there is no significant different on performance.

        PyObject *bytes = PyBytes_FromStringAndSize("abc", 3);
        if (bytes == NULL) {
            return NULL;
        }
        Py_DECREF(bytes);

and:

        PyObject *bytes = PyBytes_FromString("abc");
        if (bytes == NULL) {
            return NULL;
        }
        Py_DECREF(bytes);

Add bytes_alloc() helper function. bytes_alloc() and _PyBytes_FromStringAndSize() have less conditional branches than previous code, and so should be a little bit faster. * Rename existing bytes_alloc() to bytes_type_alloc(). * Replace _PyBytes_FromSize(calloc=1) with _PyBytes_FromSizeZero(). * Add set_ob_shash_unsafe(): similar to set_ob_shash() but don't use an atomic operation on Free Threading. Use this new function on newly allocated bytes objects and in bytes_resize_inplace(). * Replace PyBytes_FromStringAndSize(NULL, size) with bytes_alloc(size). * Use bytes_alloc() in PyBytes_FromString() and _PyBytes_Repeat(). * Add _PyBytes_FromStringAndSize(): similar to PyBytes_FromStringAndSize(), but str must not be NULL. * Replace PyBytes_FromStringAndSize() with _PyBytes_FromStringAndSize(). * Test that PyBytes_FromString() and PyBytes_FromStringAndSize() return singletons for 0 or 1 bytes.

Previously, it creates an empty bytes string which became the empty string singleton.

vstinner · 2026-10-03T20:54:21Z

Ooops, I pushed "Add static inline _Py_NewReferenceInline() function" change by mistake. This time, I used git push --force to remove this commit. It shouldn't be part of this PR.

eendebakpt · 2026-10-03T20:56:40Z

@vstinner Auto merge?

eendebakpt · 2026-10-03T20:58:52Z

There are some corner cases that hit an assert in debug builds. Reproducer:

import sys

class B(bytes):
    pass

def rt(x):          # keep the compiler from constant-folding
    return x

empty = rt(b'')
print(sys.version)

cases = {
    "B() + B()":                  lambda: B() + B(),
    "B() + bytearray()":          lambda: B() + bytearray(),
    "B() + memoryview(b'')":      lambda: B() + memoryview(b''),
    "b'' + B()  (control)":       lambda: empty + B(),
}
for name, fn in cases.items():
    r = fn()                      # debug build: aborts here with
                                  # "Assertion failed: size >= 1 ... bytesobject.c"
    print(f"{name:28} -> {r!r:6} is b'': {r is empty}")

Note: returning the singleton more often is a nice-to-have, if we return another empty bytes in some cases that would be fine for me.

new_size cannot be 0 in practice

vstinner · 2026-10-03T21:28:08Z

@vstinner Auto merge?

I consider that the PR is ready to be merged, so yes, I enabled auto merged. After I saw your comment with a regression, I disabled auto merge.

There are some corner cases that hit an assert in debug builds. Reproducer: (...)

Oh, well spotted! I misread if (va.len == 0 && PyBytes_CheckExact(b)) { in _PyBytes_Concat(), I missed the PyBytes_CheckExact() part of the condition.

It seems like this case wasn't tested at all by the Python test suite. I fixed bug and I added tests (test_concat_cpython).

Note: returning the singleton more often is a nice-to-have, if we return another empty bytes in some cases that would be fine for me.

The PR only changes bytes * int to return the empty string singleton if the number is zero. It's more a side effect of the implementation rather than a deliberate optimization :-)

eendebakpt · 2026-10-04T17:23:30Z

+        }
+    }
+    else {
+        result = bytes_get_empty();


What if either a or b is a subclass of bytes?

a and b types don't matter much. What matters is what PyObject_GetBuffer() returns. If len of the two Py_buffer are zero, the function returns the empty bytes singleton. The difference with Python 3.15 is that Python 3.15 creates a fresh empty string object in this case, whereas the singleton is returned with my change.

In Python 3.15, _PyBytes_Concat() always return a bytes object. My PR doesn't change this property.

Anyway, I added more tests on a bytes subclass :-)

eendebakpt · 2026-10-04T20:46:31Z

+        self.assertIs(empty + empty_subclass, empty)
+        self.assertIs(empty_subclass + empty, empty)


Suggested change

self.assertIs(empty + empty_subclass, empty)

self.assertIs(empty_subclass + empty, empty)

self.assertIs(empty + empty_subclass, empty)

self.assertIs(empty_subclass + empty, empty)

self.assertIs(empty_subclass + empty_subclass , empty)

Ah right, I didn't test empty_subclass + empty_subclass case. Well, I just merged my PR and I don't think that it's worth it to make a change just to add this case. It's not easy to cover all cases, but this PR extends the tested cases at least :-)

vstinner · 2026-10-04T21:45:53Z

I merged my PR. Thanks for your review @eendebakpt!

vstinner added the skip news label Oct 3, 2026

bedevere-app Bot mentioned this pull request Oct 3, 2026

C API: Rework and enhance the PyBytesWriter implementation #158585

Open

bedevere-app Bot added the awaiting core review label Oct 3, 2026

vstinner requested review from ZeroIntensity and ericsnowcurrently as code owners October 3, 2026 20:50

vstinner added 4 commits October 3, 2026 22:51

Fix test_tracemalloc.test_set_traceback_limit()

15adae3

Previously, it creates an empty bytes string which became the empty string singleton.

Add bytes * 0 singleton test

b67b770

Complete test_repeat_singleton()

1a7d8d5

vstinner force-pushed the writer_create branch from 4335acb to 1a7d8d5 Compare October 3, 2026 20:53

vstinner enabled auto-merge (squash) October 3, 2026 20:54

vstinner disabled auto-merge October 3, 2026 20:59

vstinner added 2 commits October 3, 2026 23:16

Fix _PyBytes_Concat()

ca9451c

Update byteswriter_resize()

b182597

new_size cannot be 0 in practice

eendebakpt reviewed Oct 4, 2026

View reviewed changes

vstinner added 2 commits October 4, 2026 22:17

Add tests on a bytes subclass

fa27190

Merge branch 'main' into writer_create

c1d34b2

vstinner enabled auto-merge (squash) October 4, 2026 20:31

eendebakpt reviewed Oct 4, 2026

View reviewed changes

vstinner merged commit 8dba039 into python:main Oct 4, 2026
51 checks passed

vstinner deleted the writer_create branch October 4, 2026 21:01

bedevere-app Bot removed the awaiting core review label Oct 4, 2026

Navigation Menu

Uh oh!

gh-158585: Optimize bytes object creation - #158664

gh-158585: Optimize bytes object creation#158664
vstinner merged 8 commits into
python:mainfrom
vstinner:writer_create

vstinner commented Oct 3, 2026 •

edited by bedevere-app Bot

Loading

Uh oh!

vstinner commented Oct 3, 2026

Uh oh!

vstinner commented Oct 3, 2026 •

edited

Loading

Uh oh!

vstinner commented Oct 3, 2026

Uh oh!

vstinner commented Oct 3, 2026

Uh oh!

eendebakpt commented Oct 3, 2026

Uh oh!

eendebakpt commented Oct 3, 2026

Uh oh!

vstinner commented Oct 3, 2026

Uh oh!

eendebakpt Oct 4, 2026

Uh oh!

vstinner Oct 4, 2026

Uh oh!

eendebakpt Oct 4, 2026

Uh oh!

vstinner Oct 4, 2026

Uh oh!

Uh oh!

vstinner commented Oct 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		self.assertIs(empty + empty_subclass, empty)
		self.assertIs(empty_subclass + empty, empty)

Uh oh!

Conversation

vstinner commented Oct 3, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Oct 3, 2026

Uh oh!

vstinner commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vstinner commented Oct 3, 2026

Uh oh!

vstinner commented Oct 3, 2026

Uh oh!

eendebakpt commented Oct 3, 2026

Uh oh!

eendebakpt commented Oct 3, 2026

Uh oh!

vstinner commented Oct 3, 2026

Uh oh!

eendebakpt Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

vstinner Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

eendebakpt Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

vstinner Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vstinner commented Oct 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vstinner commented Oct 3, 2026 •

edited by bedevere-app Bot

Loading

vstinner commented Oct 3, 2026 •

edited

Loading